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The invention relates to a system for operating with different types of media 
contend the system being arranged to enable a user to use a first content of a first type. 

The invention also relates to a method of operating with different types of 
media content, the method comprising a step of identifying a user's usage of a first content of 
a first type. 



Document US2002/0002564A1 discloses data processing means for viewing, 
playback and delivary of continuous audio-visual data, media content such as a motion 
picture, video program, audio program etc. The media content, e.g. MPEG-1 system stream 
or in other format, and description data describing audio and video content are received by 
said means from outside, or said content and data can be stored in a database fix>m which they 
can be retrieved. The description data are expressed through use o^ e.g„ Extensible Markup 
Language (XML) whose standardization is pursued by the World Wide Web Consortium. 
The description data describe parameters of the audio and video content such as a title, 
caption, priority, location, start time, end time etc. Video and audio streams of the media 
content are mutually synchronized and provided to video playback means and audio playback 
means. The media content is further rendered for the user. In the known data processing 
means, the user may request or disable simultaneous rendering of the provided audio and 
video contents. The user may choose to be provided with only the audio part or only tihe 
video part of the available media content. 

In the known data processing means, the audio and video contents correspond 
to each other since they are received together from a content provider, e.g. a TV broadcaster 
or the like. At the same time, the user may like simultaneous rendering of media contents 
which are not related by the content provider. It is a drawback of tiie prior art that user's 
preferences with respect to rendering the different contents are not taken into account. 
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It is an object of the present invention to provide a system for operating with 
different types of media content of the kind defined in the opening paragraph which does take 
into accomit the user*s preferences with respect to simultaneous rendering of the different 
cont^ts. 

5 The object of the invention is realized in that the system comprises identifying 

means for identifying that the user concurrently uses a second content of a second type» said 

secoBd^gtent^^eing-AJBrelated^aatb^ as soc iatin g m eans-for.assoc i ating — 

said second content with the first content. 

The contents of different types, e.g. audio and video, which are not associated 

10 by any reference in any way in the system may be related in the user^s mind. For example, 
the user may like simultaneous rendering some contents of the different types, or the 
combination of the contents may carry the user back to some life experience, or the like. The 
system comprises identifying means and associating means for "registering" the relationship 
between such contents in the system. The identifying means identifies that the user uses, e.g. 

1 5 watches, listens to, etc, a first content of a first type and, may be in a different way, a second 
content of a second type at the same time, wherein said first and second contents are not 
related in the system. The associating means associates said first and second contents to 
enable the system for exploiting that relationship. 

One of the advantages of the present invention is that the user does not have to 

20 explicitly instruct the system to relate such contents because it may be done automatically 
and transparently for the user without hindering the user's usage of said contents. Another 
advantage is that the links between contents are made personal to the user of the system, 
which is not be possible when the contents, e.g. a TV program, movie which usually include 
audio and video, are obtained from the content provider. 

25 • The object of the present invention is also realized in that the method of 

operating with different types of media content further comprises a stqp of identifying that 
the user concurrently uses a second content of a second type, said second content being 
unrelated with the first content, and a step of associating said second cont^t with the first 
content. The method describes functioning of the system of the present invention. 

30 



These and other aspects of the invention will be further elucidated and 
described with reference to the accompanying drawings, wherein: 

Fig. 1 illustrates the general scheme of the present invention; 
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Fig. 2 shows a functional block diagram of the system suitable for 

implementing the present invention; 

Fig. 3 shows an example of the associated contents of different types and the 

relationship between them; 

Fig. 4 shows an example of a time sequence of the user's usage of different 
media contents; 

Fig. 5 shows an embodiment of the method of the present invention. 
Throughout the figures, same referaice numerals indicate identical or 
corresponding compon^ts. 



Many people today use a lot of media content. People watch, listen, browse 
different audio and video information, as well as CTeate their own media content, e.g. by 
voice recording, taking pictures using a camera, making a home fihn etc. Various consumer 
electronic devices 110 such as a television set (TV set) with a cable, satellite or other link, 
video cassette recorder (VCR), home stareo, internet broadcasts, world wide web, portable 
and stationary radio, CD player, pasonal digital assistant (PDA) etc can be used for that 
purposes, as it is shown in Fig. 1 . Each of these devices may have a user interface used to 
obtain, select, record, play etc the media content. Any one of such consumer electronic 
devices allows the usct to use at least one media content, and some of such devices may 
output or input at least two contents of different types. The system according to the present 
invention comprises identifying means 120 for identifying that the user concurrently uses at 
least two contents of different types, whereas that contents are not unrelated in the system, 
e.g. the s>«tem does not include data indicating the relation between said contents. Such 
identifying means 120 may be any means enable of monitoring user's activities in the 
system, e.g. a monitoring camera cormected to a devoted data processing unit which is 
arranged for analyzing the user's behavior by processing data obtained &om the camera, said 
data processing unit may be a part of the consumer electronic device rendering the media 
content. 

One or many media devices capable of outputting and/or inputting the media 
content may be included into the system. When a plurality of such devices are included, they 
may have means (not shown) for conmnunicating with each other, e.g. the devices may be 
connected into a network. The communication means may be arranged for obtaining 
information about the user's usage of the content j&om each of the devices, and recognizmg 
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individual devices providing said infonnation. For example, each device outputs to the 
communication means its identification and information pertaining to the contents used by 
the user, e.g. the type of the used content(s) which may be one of predetermined types in the 
system, etc. Further details which concern functioning of the identiLfying means will be 
S apparent to the person skilled in the art. 

The media content may basically comprise at least one or any combination of 

visualiafeEmation,-^udio4n&«mali^ 

reproduction, or the like. The expression audio data, or audio content, is hereinafter used as 
data pertaining to audio comprising audible tones, silence, speech, music, tranquility, external 

10 noise or the like. The expression video data, or video content, is used as data which are 
visible such as a motion picture, static image, characters etc. The user device may be 
arranged to obtain the media content stored on different carriers such as, audio tapes, video 
tapes, optical storage discs, floppy and hard drive discs etc in any format, e.g. MPEG 
OVIoving Picture Experts Group), MIDI (Musical Instrument Digital Interface), Shockwave, 

1 5 QuickTime, WAV (Waveform Audio), etc. The content may also be broadcast by a content 
provider, e.g. TV or radio, to be received by the user devices rendering said content for the 
user. The user may use the media contCTt at home, public places, at the public transport and 
so on. Examples of the different types of content may be a play-list of songs or music, 
moving picture, still image or set of the still images, e.g. an image slide, animation, text, 

20 meta-data related to the media content etc. For instance, the meta-data may define the radio 
content, e.g. radio stations and their the most frequently broadcast content, that is available in 
the area of user's living, the different kinds of playlists for setting up a collection of the 
media contents. The meta-data may, for example, include the music playlist, e.g. in which the 
next song is started when the current one ends, the radio station or television channel playlist, 

25 e.g. comprising a collection of favorite channels to watch or surf through. 

With reference to Fig. 2, a functional block diagram of the system of the 
present invention is shown. The system 200 comprises the identi^dng means 120 and 
associating means 210 for associating said two or more contents simultaneous use of which is 
identified by the identifying means 120. The identifying means outputs to associating means 

30 the information concerning the contents concurrently used in the system. The associating 
means may be arranged for establishing data which link said associated contents, such as 
meta-data comprising the information pCTtaining to a relationship between the associated 
contents. In one example, the meta-data may include infonnation about the associated 
contents such as a name/title, source of or link to the contents, e.g. identification of the 
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device(s) used for rendering the content; observed user's behavior during the use of the 
content(s), e.g. types of the user's behavior like watching, listening etc may be 
predetermined, whether or not the user initiated the content rendering, date, start and/or end 
time of the content usage, duration of rendering etc as it is shown in Table 1. Of course, some 
5 of Ihe information included into the meta-data may be obtained from the identifying means 
120. 



TABLE 1 





Name/title 


Source/link 


User's 

behavio 

r 


Initiat 
ed by 
user 


Start 
date/ 
time 


Durati 
on 


Rati 1 
ng 


n-l 
















n 


"Content 
A" 


PC: D:\clips\ 
clipN 


Watchi 


Yes 


07/07/ 
02 

01:30p 
.m 


00:29: 
37 


0,5 


"Content 
B" 


Radio: song^ 


Disreea 
rd 


No 


07/07/ 
02 

01:36p 
.m 


n+l 


"Content 
C" 


PC: 

C:\video\fragment 
N 


Watchi 
ng 


Yes 


09/07/ 
02 

06:12p 
.m 


00:12: 
13 


1,0 


"Contaat 
D" 


CD-player: 

Disc"AlbumA'^ 

trackN 


Listeni 
ng 


Yes 


09/OV 
02 

06:06p 
.m 


n+2 

















10 The associating means may be arranged to associate the contents automatically 

or to commxmicate information pertaining to the contents to the user for obtaining his/her 
approval for associating those contents. The system may also simply notify the user about 
associating the corresponding contents. 




23.09.2002 



The system may optionally comprise storage means 220 arranged for storing 
the meta-data, selection means 230 arranged for selecting the content, output means 240 
arranged for outputting the associated contents, and rating means 250 arranged for rating the 
associated content. 

5 The meta-data may be implemented in any standard data structure or database, 

any language, e.g. XML, and include any amount of information about the associated 

contents which is sufficient to distinguish them. In one example, the meta-data may be stored 

in the storage means 220 together with the corresponding contents, e.g. in a header of the 
corresponding media data. The new meta-data may be established each time when some 

10 contents are associated by the associating means. Thus, a pluraUty of meta-data records may 
be stored by the storage means. 

Given meta-data storing information about two associated contents, the 
selection means 230 may be arranged to identify the first content upon selection of the 
associated second content, and vice versa. Thus, the system can always find all associated 

1 S contents if one of them is selected and the relationship between those associated contents has 
been established. The output means may be arranged to play, reproduce etc the associated 
contents simultaneously. 

The selection means may be further arranged to fimction as a recommender of 
the media content. The recommender may exploit the meta-data for recommending the 

20 content in accordance with the user's preferences. In that way, the recommendations can be 
made personal to the user. 

The recommender may be arranged for recommending the associated contents 
if the user selects one content and likes to be recommended with another content of different 
type in accordance to his/her preferences. The recommender may be fiirther arranged for 

25 recommending contents correlating to one of the associated contents e.g, having similar 
subject, topic, category, genre etc. For example, if tihie song and clip are associated, the 
reconmaender may recommend, i^on the selection of the clip, the associated song and other 
songs which are similar to the associated song. 

The recommender may be arranged to recommend the content on the basis of 

30 rating provided by the rating means 250. The meta-data may be interpreted in different ways 
for rating them, e.g. as an explicit or implicit feedback from the user. For example, the media 
contents may be considered as explicitly associated by the user if the user has explicitly 
initiated reproduction, or the other way of a user^s consmnption, of both media contents, e.g. 
user's requests for rendering the contents were inputted to the system. The rating means 250 
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may highly rate such association of the contents, or a high reliability may be assigned to such 
association. As it shown in Table 1, the QST+l) meta-data record indicates that the user 
initiated displaying the video content C and playing the audio content D. Therefore, flie 
contents C and D axe associated with the maximum rating, for example with one point as a 
5 maximal possible rating. If one of the contents was not explicitly chosen for rendering or 
using it otherwise by the user, then the association of that contents may be rated lower or 
given less reliability. For example, in Table 1 the contents A and B are associated with a 
rating lower than the rating of contents C and D, namely with 0.5 point out of 1 , because the 
content B, a song, was played on the radio, and the user did not initiate that playing. 

10 OthCT criteria for evaluating the association of the contents may be used by the 

rating means. For example, the association of the media contents may be also influenced by 
the user's behavior during simultaneous rendering of the content, a user's reaction for 
initiating simultaneously rendering of the contents when it was started by the system itself 
and so on. Other parameters and characteristics of the associated contents, user's usage of the 

1 5 associated contents etc may be stored in the meta-data and used for their rating. 

In a further embodiment, the rating means may rate the associated contents 
depending on the number of times of using the associated contents simultaneously. For 
instance, if the contents were often used together, they are rated highly. 

In a further embodiment, if ttie first content is associated with the second 

20 content and with a third contmt. Hie rating means may be arranged for rating said 

associations of the first content with the second content, and the first content with the third 
content differently. For example, the association of the first content and second content may 
be rated higher if the first content was more often used with the second content than with the 
third content. In another example, the association of tiie first content may be rated higher 

25 with the second content if the simvdtaneous usage of the first and second contents is more 
recent than the usage of the first and third contents. Thus, such associated contents are rated 
higher if they were recently used. 

An example of associated contents is shown with reference to Fig. 3. Content 
"audio 1" 310 is associated with content 'Mdeo 1" 320, such association iUustrates the idea 

30 of the present invention about association of the contents of different types. At the same time, 
content •^ddeo 1" 320 maybe further associated with the other contents of different type, e.g. 
"audio 2" 330, "audio 3" 340 etc. The type of content "music playlist" 350, which is also 
associated with content "video 1" 320, maybe considered as a separate type of the content 
corresponding to the group of contents of type "music". In turn, contents "audio 1" 310 and 
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"audio 3" 340 maybe associated with content "image 1" 360 and content *Video 320. In 
this example, if the user selects content "audio 3" and requests the reconmtiender to 
recomnaend the media content of type different from type "audio", the recommender may 
recommend contents "image 1" 360 and * Video 1" 320 since they are associated with the 
5 user-selected content "audio 3" 340. 

With reference to Fig. 4, an example of media contents used at different time 

in the system is shown. ^TMedia content 1" 410 was used in the system from time T1 1 (41 1 ) 

till time T12 (412), "media content 2" 420 - from time T21 (421), "media content 3" 430 - 
from time T31 (431). Contents 410, 420 and 430 have different time length of their usage. 

10 The usage of content 410 was started earlier than the usage of content 420, though this 

difference in time (T21-T1 1) is much less than the duration (412-421) of the simultaneous 
usage of contents 410 and 420. The identifying means 120 may be arranged to provide a 
starting time, ending time, time of simultaneous usage of the contents to the associating 
means 210 for including said time information into the corresponding meta-data. The 

1 S associating means may be arranged to estimate a ratio between duration of simultaneous 
usage (412-421) of the corresponding contents and time difference (T21-T1 1) or difference 
of times when the use of corresponding contents ends. In one example, the associating meam 
may be arranged to ignore the simultaneous usage of the contents and the respective meta- 
data may not be established if the duration of simultaneous usage of them is much smalls 

20 than a tiheir non-overlapping in time usage of them. For example, the duration (412-431) of 
simultaneous usage of contents 410 and 430 is rather small with respect to the duration of 
their non-overlapping in time usage. The associating means may include a value of said 
estimated ratio into the respective meta-data. In another example, the rating means may rate 
the association of contents basing on said ratio. For instance, the association of contents 410 

25 and 420 may be rated higher than the association of cont^ts 410 and 430 in view of that the 
considerations above. 

In one of the embodiments of the present invention, the selection means 230 
may enable the user to modify the meta-data in flie system. Of course, an appropriate user 
interface including input means, e.g. keyboard^ mouse, touch-screen, speech recognition 

30 means etc, may be provided, and the output means 240 may be arranged to present the meta- 
data to the user via, for instance, display device, audio reproduction means etc. 

The identifying means, associating means, selection means, rating means may 
be implemented using a microprocessor (not shown) coupled to random access memory and 
read-only memory (ROM) storing a program which when executed by said microprocessor 
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can perform functions of said means as described above. The storage means 220 may be 
realized with said ROM. The media content may be stored locally in the system or 
downloaded from an external source by means of suitable arranged data receiving means (not 
shown). The realization of such microprocessor system with corresponding internal and 
5 external circuits will be apparent to the person skilled in the art and need not be discussed 
herem. 

With reference to Fig. 5, the method of the present invention describing 
operation of the system of the present invention is shown. According to the method of 
operating with different types of media content, the concurrent usage of the jSrst content and 

10 second contents of different types is identified at step 510, wherein the relationship of said 
contents is not known in the system. At step 520, said first and second contents are associated 
as being related by the simifltaneous user's usage of them. For instance, liie user may watch a 
slide show of pictures on the display screen together with a certain song being played by the 
CD player in the system, or the user may often listen some song when the user would like to 

15 download certain pictures. Such contents being used by the user simultaneously are 
associated. Embodiments of the method, which correspond to the described above 
embodiments of the system of the present invention, may be envisaged. 

The various program products may implement the fimctions of the system and 
method of the present invention and may be combined in several ways with the hardware or 

20 located in different devices. Variations and modifications of the described embodiment are 
possible within the scope of the inventive concept. Thus, for example, the use of the verb 'to 
comprise' and its conjugations does not exclude the presence of elements or steps other than 
those defined in a claim. The invention can be implemented by means of hardware 
comprising several distinct elements, and by means of a suitably programmed computOT. In 

25 the device claim enumerating several means, several of these means can be embodied by one 
and the same item of hardware. 

A 'computer program' is to be imderstood to mean any software product 
stored on a computer-readable medium, such as a floppy-disk, downloadable via a network, 
such as the Internet, or marketable in any other maimer. 
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CLAIMS: 



EPO-DG1 
24. 09. ^002 



EPO - L J; 1 

1 . A system for operating with different types of media content, the system being 
arranged to enable a user to use a first content of a first tjpe, characterized in that the system 
comprises: 

identifying means for identifying that the user concurrently uses a second 
content of a second type, said second content being unrelated with the firat content, and 

associating means for associating said second content with the first content. 

2. The system of claim 1, further comprising storage means arranged for storing 
meta-data comprising information pertaining to said associated first and second content. 

3. The system of claim 1, further comprising selection means arranged for 
selecting the content. 

^- The system of claim 2 and 3, wherein said selection means are further 

airanged to identify the first content upon selection of the associated second content and/or to 
identify the second content upon selection of the associated first content, using said 
information stored in the meta-data. 

5. The system of claim 4, wherein said selection means are fijrther airanged to 

fimction as a recommender for recommending the associated first or second content upon a 
user-operable selection of one of said associated second and first content respectively, using 
said selection means. 



6. The system of claim 4, further comprising output means airanged for 

simultaneous outputting said associated first and second contents. 

The system of claim 2 and 3, wherein said selection means are further 
arranged for user-operably modifying said meta-data. 
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8. The system of claim 1, wherein said identifying means is arranged to identify 

a user's usage of a third content of a second or other type, said usage being concurrent to said 
user's usage of the first content, and said third content being unrelated with the first content, 
and wherein said associating means is arranged to associate said third and first contents, the 
5 system fiulher comprising rating means arranged for rating said association of the first 
content with the second content and/or witii the third content 



9. The system of claim 1 , comprising a plurality of devices, each device 
including output means arranged for ou^utting at least one type of the media content, and/or 

1 0 input means arranged for obtaining at least one type of the media content. 

10. The system of any one of the preceding claims, wherein said first and second 
content correspond to video and audio content. 

15 XL A method of operating with different types of media content, the method 

comprising a step of identifying a user's usage of a first content of a first type, characterized 
in that the method fiirther comprises 

a step of identifying that the user concurrently uses a second content of a 
second type, said second content being unrelated witii the first content, and 

20 - a step of associating said second content with the Gist content. 



12. A computer program product enabling a programmable device when executing 

said computer program product to fimction as the system as defined in claim L 
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The invention relates to a system for operating with different types of media 
content, the system bemg arranged to enable a user to use a first content of a first type. The 
system comprises identifying means (120) for identifying that the user concurrently uses a 
second content of a second type, said second content bemg unrelated with the first content, 
and associating means (210) for associating said second content with the first content. The 
invention also relates to the method of operatmg with the different types of media content. 
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